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Overview 

q  Descri  ption  of  the  experience  and  lessons-leamed  as  a  result  of 
applying  the  T ri  Service  Assessment  process  to  an  acquisition 
organization  within  NAVSEA  (PEO  IW5  nfe  PEOTSC) 

I  W5  =  Integrated  Vteirfare  Systems 

q  Focus  on  the  specific  challenges  encountered 

and  the  process  adaptations  made  to  ensure  the  assessments  vere 
successful 

q  These  recorrmendations  are  pragmatic 

Tips  to  make  things  better 

q  This  briefing  does  not  summarize  the  technical  findings 

but  rather  focuses  on  howthey  v\ere  achieved 
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Wny  assessments? 

q  At  certain  times,  for  certain  progams,  assessments  arc 
necessary  to 

Check  for  latent  surprises 
Accelerate  process  irrprovement  planning 
Identify  problerrs  and  root  causes 
Facilitate  budget  and  resource  planning 
Assess  risk  for  upcoming  events  (e.g.,  OPEVAL) 

q  Use  of  assessments  has  proved  to  be  a  valuable  tool  to  assist  in 
acquiring  software- intensive  systems 

q  But  assessments  must  be  carefU  ly  conducted  to  achieve 
maximum  utility 
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WryTAI  process  was  selected 

q  Key  features 

Pne-defined,  v\ell-used  process 

Network  of  independent  experts  available  to  serving  on  assessment  teams 
Cross-service  perspective  (“How did  they  solve  it?”) 

q  Overall  goal  for  conducting  assessments: 

Evaluation  and  irrpraMerrent  of  the  software  processes  used  by  programs 
vuthinthe  PEO 

Identification  of  vveaknesses  and  strengths 
Determination  of  risk  and  risk  mitigation  strategies 

Establ  ishmert  of  sofbAare  guidance  to  raise  level  of  development  qual  ity 
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PLrpose  of  assessments 

q  Prepare  for  CEC  OPEVAL  by  ensuring  no  unpleasant  surprises 
v\ere  lurking  -  assess  the  five  core  systems  that  use  CEC  as  a 
means  for  track  shari  ng 

»  CEC  -  Cooperati\/e  Engagement  Capability 
»  E2C  -  NAVAIR  program 
»  ACDS  -  Carrier  Self  Defense  System 
»  Aegis  6  Phase  1 
»  C2P  -  Corrrmnd  Control  Processor  -  SPAVNAR  -  PMW 159 

q  Three  separate  organizations  are  represented  among  these 
systems  (NAVSEA,  NAVAIR,  and  SPAWAR) 

q  Four  of  the  five  programs  v\ere  assessed 

q  (Subsequently,  an  additional  six  programs  have  been  assessed 
within  the  PEO) 
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Wratwe  learned... 


q  The  benefits  to  performing  assessments  are  substantial 
Clear  and  timely  snapshot  of  current  state  of  program 
Roll-up  of  lessons-leamed 
Direct  assistance  to  the  programs 

But , 

q  Unless  carefully  performed,  these  benefits  may  be  difficult  to 
achieve 

q  There  exist  several  pitfalls  that  can  derail  ability  to  derive  value 
from  assessments 

q  Team  had  to  adapt  to  maximize  effectiveness 
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Pitfalls 

1  'Slow  rol l"i n  scheduling  interviews  and  site  visits 

2  Site  visits  causing  disruption 

3  Ankle  biter  reputation 

4  'Fire  hose"  site  visits 

5  Missing  important  topics 

6  ‘SW  only"  assessment 

7  Extended  assessment  schedule 

8  High-lewel  and  process-only 

9  'Dump  and  run1 1  syndrome 

10  Multiple,  unrelated  issues 

11  Late  final  reports 

12  Unclear  presentation  of  findings 

13  Criticismcausing  defensiveness 

14  Recommendations  requiring  excessive  time 
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1-  'Slow  roll"  in  scheduling  interviews  and  site  visits 

q  Being  the  subject  of  an  assessment  is  never  fun 

Like  going  to  the  dentist 

q  Tendency  is  to  delay  it  as  long  as  possible 

Blaming  it  on  coordinating  schedules 

q  Tasks  that  are  irrportant;  hovever,  seemto  alv\ays  get  done 
q  Need  to  ensure  that  everyone  has  sane  sense  of  priority 
q  Recomrendations: 

Secure  assurances  from  PM  that  assessment  is  of  high-priority 

Ensure  that  PM  makes  this  clear  to  organizations  being  assessed 

Assessment  team  members  need  to  be  flexible  to  accommodate 
reasonable  schedule  changes 

Emphasize  that  very  little  staff  time  is  really  needed  (see  =t£>) 

Define  a  schedule  and  follow  it 
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2  -  Site  visits  causing  disruption 

q  In  all  cases,  encountered  significant  push-back  from 

organizations  being  assessed  -  fear  of  disruption  and  delays 

Fora  program  in  trouble,  impact  could  be  serious 

q  Disrupting  the  progress  of  the  program  by  performing  the 
assessment  is  counteiproductive 

Often  create  more  problems  than  those  solved 

q  Need  to  ensure  a  ripple-free  process 

and  to  set  their  minds  at  ease  i n  advance 
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2  -  Site  visits  causing  disruption  (contd) 

q  RecorTrrendati  ons : 

Schedule  each  person  for  no  more  than  1  hour  for  entire  visit 

»  Some  incSvi duals  may  require  longer  times 

>  POC,  guides,  etc... 

>  Keep  to  a  rrinirrum 

Plan  interviews  based  on  organization  chart-  functions  and  people 

»  Rather  than  by  topic 

Use  this  approach  to  assess  corrrruni cati ons  across  organization 

»  A  key  problem  in  many  programs 

Request  that  no  special  rrateri al s/bri efi  ngs/etc.  be  prepared  for  the  site 
visits 

»  Ensure  fidelity  of  snapshot  -  rrinirrize  Heisenberg  effect 
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Staff 


Recomrended  i  rterview  schedule  approach 


Easier  for  interviewers 


Easier  for  interviewees 


Topics 


Topics 
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3  -  Ankle  biter  reputation 


q  Typical  ly,  programs  react  vuth  fear  that  an  assessment  is  coning 

"Not  more  help!” 


q  If  the  assessors  leave  the  impression  that  they  are  there  just  to 
criticize,  the  staff  vu  1 1  be  uncooperative 
Crucial  information  vull  not  be  forthcoming 
and  the  vvond  vull  spread 

q  Recommendations: 


During  interviews 

»  Always  be  supportive 

»  J  ust  collect  inforrration,  make  no  judgments 
»  Encourage  dialog,  let  them  talk 


During  briefouts 


» 


Ensure  balance  of  find ngs  (favorable  and  unfavorable) 
Emphasize  that  all  procparrs  have  roomfor  improvement 
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4  -  'Fire  hose"  site  visits 

q  Lots  cf  data  is  presented  during  interviews 
q  The  data  is  not  organized  according  to  topic  (see  #2) 
q  Much  is  repetitive 

q  This  "fire  hose"  situation  may  result  in  information  being  lost 

Need  to  organize  and  assimilate 

q  Recommendations-. 

Each  team  member  should  take  note  of  findings  and  tentative  conclusions 
as  interviews  progress 
»  Don't  wait  until  later 

Leave  15  rrin  or  so  between  witnesses  to  allow  for  soak-in  time  and 
discussion 

Teams  should  caucus  privately  at  least  once  per  day  to  collect  thoughts 
and  compare  fi  ndi  ngs 
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5-  Missing  inportant  topics 

q  Assessments  that  do  not  touch  all  important  topics  rray  rriss 
critical  issues  and  may  tail  to  reinforce  good  practice 

q  Too  rruch  time  spent  on  ensuring  total  coverage  vull  skimoxer 
key  topics  and  not  permit  sufficient  depth 

q  Recommendations: 

UsetheTAI  infbrrmtion  model  to  helpstnjcture  topics 
Make  a  list  beforehand  that  includes  relevant  topic  areas 
»  Assign  team  members  to  ensure  topics  are  covered 
Rely  on  team  experience  to  determine  topics  to  cover  and  skip 
Ask  interviewees  what  their  view  of  important  topics  is 
»  Often  those  closest  to  the  fire  know  where  the  heat  is 
»  Be  wiling  to  stroll  down  unexpected  paths 
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6-  'SW only"  assessment 

q  If  advertised  as  a  'SW only"  assessment  the  results  may  be 
ignored  as  being  irrelevant 

'It  doesn't  affect  me  -  I  do  systems." 

q  If  I i rrited  to  a  'SW  only"  assessment  the  results  may  actual  ly  be 
irrelevant 

Rarely  is  SWthe  prime  cause  of  problems,  although  a  common  perception  is 
that  SW  is  vN/here  the  problems  reside 

q  Recommendations-. 

Be  clear  about  the  assessments  focus  on  softvxoe 

»  "...plus  all  factors  that  affect  the  ability  of  the  SW  developers  to  do  their 
job" 

Ensure  that  non-SW factors  are  considered  in  the  assessment 
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7  -  Extended  assessment  schedule 

q  An  extended  assessment  schedule  will 

Dilute  findings 

Likely  to  be  inaccurate  as  programevolves 
Have  reduced  irrpact 

q  Often  this  is  a  big  challenge 

Blending  schedules  is  often  very  hard 

Unwillingness  to  prioritize  assessment  may  be  a  finding  itself 

q  Recommendations: 

Keep  site  visits  to  within  a  v\eek  max  tor  each  site 

Emphasize  importance  of  keeping  things  moving 

Be  wiling  to  proceed  with  subset  of  assessment  team 

Be  willing  to  be  flexible  about  missing  some  key  players 

Do  not  let  activity  stretch  out-  place  high  priority  on  getting  it  done 
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8  -  High-leMsI  and  process-only 

q  Assessment  reports  that  are  high-level  and  focus  solely  on 
process  are  likely  to  be  viev\ed  as  being  impractical 
and  may  very  v\ell  be 

q  Ignoring  the  details  can  blur  Linderstanding  the  big  picture 

q  Hand-v\aving  level  not  likely  to  result  in  real  change 

Too  tar  removed  from  worker-level 

q  Yet  often  hic^rlevel  recomTrendations  have  the  most  lasting 
impact 

q  Recommendations: 

Ensure  low-level  information  is  collected  and  appropriately  reported 

Correlate  process  findings  wth  objective  artifacts 

Be  specific  about  findings  and  recommendations  -  use  details  to 
demonstrate  global  issues 
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9  -  'Durrp  and  run"  syndrome 

q  Characterized  by  teanis  delivering  final  report  to  PM  and  then 
disappearing 

q  V\/ll  not  have  a  lasting  irrpact 

Reports  often  need  additional  explanation 

Recommendations  often  need  cultivation  and  continued  attention 
Results  need  to  be  briefed  to  PO  and  Contractor  staff 

q  Recomrmrdations: 

Do  not  disappear  after  presentation  of  final  report 

Encourage  PM  to  allow  report  to  be  presented  to  developers  /contractors/ 
labs 

Create  Transition  Plan  to  ensure  continued  support 
»  Make  some  subset  of  team  available  afterv\ords  for  specific  guidance 
»  Transition  to  dedicated  teamfor  irrplerrentation 
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lO-  Multiple,  unrelated  issues 

q  Reporting  rrultiple,  unrelated  issues  vull 

Present  a  confused  picture  of  the  state  of  tine  program 
Delay  being  able  to  identity  root  causes 

q  Need  cohesive  vievvs  to  see  totality  of  program 
q  'In  the  v\eeds'' perspective  can  rriss  irrportant  lessons 
q  Recommendations : 

Ensure  team  has  sufficient  time  to  rrull  over  findings  and  observations 
Look  for  common  causes  and  fixes 
Categorize  findings  into  logical  bins 
Analyze  cause-effect  relationships 

Organize  according  to  recommendations  rather  than  symptom 
»  Focus  on  proactive  solutions 
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11  -  Timely  final  reports 

q  Once  a  program  is  convinced  that  an  assessment  is  a  good  idea, 
they  become  anxious  to  get  results 

q  For  optimum  benefit;  final  reports  are  needed  promptly  so  that 
recommendations  can  be  put  into  place 

q  Delayed  reports  are  likely  to  be  ignored,  and  may  have  little 
practical  impact 

q  Recommendations 

Brief  sites  on  initial  observations  before  leaving 
»  But  do  not  brief  any  conclusions  of  recorrrrendations 
Prepare  report  as  a  briefing 
»  Formal  narrative  reports  take  too  rruch  time 
Ensue  all  parties  are  briefed  as  soon  as  possible 
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11  -  Timely  final  reports  (contd) 

q  Recommendations  (cont’d): 

Do  not  include  all  findings  and  conclusions  -  prioritize 
Focus  on  high-leverage  recommendations 

Do  not  obsess  about  providing  all  details  -  consider  the ' 'el evator" '  speech 
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12-  Unclear  presentation  of  findings 

q  A  lack  of  a  clear  presentation  of  findings  may  result  in  a 
ccnprorrised  message  and  excessive  defensiveness 

q  Can  occur  if  report  is  too  long  and  detailed 

q  Risk  that  reccrrrrendations  vull  be  rrisunderstood  and  ignored 

q  I  ncS cation  that  issues  may  not  be  fully  understood  by  assessment 
team  (worst  case  situation) 

q  Recommendations : 

Provide  adequate  caucus  time  for  team  to  review  notes  and  observations 
Ensure  that  issues  are  dearly  stated  and  are  appropriately  grouped 
»  Provide  details  as  clarification  but  as  sub-bullets 
Recorrnendations  should  be  directly  actionable 
Leave  ti  me  for  independent  review  of  report 
Continue  to  apply  the  “elevator  rule" 
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13  -  Criticismcausing  defensiveness 

q  Reports  fi  I  led  with  criticisrrB  I  i  kely  to  result  i  n  a  defensive  posture 
by  the  program  being  assessed 
Yet;  criticisrrB  are  unavoidable 

q  Briefing  such  reports  will  degrade  into  arguments  and  sides 
being  taken 

q  Impact  is  loss  of  audence  and  failure  of  assessment 

q  Recommendations-. 

Balance  the  criticisrrB  with  praise  (not  arbitrary  but  justified) 

Errphasize  that  every  project  has  room  for  improvement 
State  clearly  that  the  results  are  just  tor  their  own  use 
Ensue  reports  are  not  personal  by  focusing  on  technical  aspects 
Sometimes,  a  harsh  message  is  necessary  however 


28J  an  2003 


23 


MITRE 


WSBaii  “Experience  and  Lessons-leamed  in  Applying  Tri-Servioe  Assessment  Initiative  Process"  Presented  at  the  Conference  on  the  Acquisition  of  Software-Intensive  Systems — January 28-30,  2003 


Sarrplefoil  to  calibrate 

I  interpretation  Of  ResUtS  reception  of  results 


q  Most  assessments  report  areas  for  irrproMement 
Nature  of  sofbAare  practice 
Everyone  can  irrprove 

q  Hav\ever,  most  organizations  do  most  things  right 

q  Many  problems  caused  by  external  factors  out  of  developers' 
control 

q  All  SW  development  organizations  have  roomfor  inproverrent 
Regardless  of  CMM  level 

q  Goal  is  to  raise  quality  level  of  software  practice 


Shoddy  process 
Very  few 


Most  common 


Best  practice 
Very  few 
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14-  Recomrerdations  requiring  excessive  time 

q  Sometimes,  obvious  and  important  recommendations  require 
time  to  implement 

Long-term  i  investment  and  payback 

q  Hov\ever,  needs  of  proclaims  often  have  shorter  time-lines 

q  Platitudes  and  generic  fixes  often  are  moot 
May  be  correct  but  irrelevant 
May  be  of  little  utility 

q  Often  tied  to  root  causes 

q  Recommendations: 
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Include  focus  on  actions  that  can  be  applied  immediately 
Ensure  that  expectations  for  benefits  are  not  exaggerated 
Admit  if  short-term  fixes  aren't  enough 
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Conclusions 

q  Use  of  assessments  can  be  a  valuable  and  cost-effective 
management  strategy 

q  Assessments  help 

to  identity  and  rritigate  risk 
to  identity  areas  for  inqDreuemert 

to  promulgate  lessons-leamed  so  that  future  programs  don't  make  the 
same  mistake 

q  To  gain  maximum  benefit  they  must  be  conducted  carefully  and 
with  proper  planning 
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